System and method for synchronizing enhancing content with a video program using closed captioning

ABSTRACT

A system and method uses a closed captioning script to synchronize supplemental data with specified junctures in a video program. A parser parses the closed captioning script to identify a set of unique key phrases. A producer associates the supplemental data with the key phrases of the closed captioning script that correspond to the desired points in the program. When the program is played, a viewer computing unit monitors the closed captioning script to detect the key phrases. Upon detection of a particular key phrase, the viewer computing unit retrieves the supplemental data associated with the particular key phrase. The supplemental data is multicast to a multicast address. A listener listens to the multicast address to receive and handle any enhancement action supported by the supplemental data. The listener can be embedded in a container, such as an HTML page or other hypermedia document.

RELATED APPLICATIONS

[0001] This application is a continuation of U.S. Ser. No. 08/779,270,filed Jan. 6, 1997, which issued as U.S. Pat. No. ______ on ______.

TECHNICAL FIELD

[0002] This invention relates to systems and methods for synchronizingenhancing content with specific junctures in a video program. Moreparticularly, the invention relates to systems and methods forsynchronizing the enhancing content with the video program by using theclosed captioning script of the video program.

BACKGROUND OF THE INVENTION

[0003] Conventional broadcast television is a non-interactive form ofhome entertainment. Television signals are broadcast one-way over abroadcast network from a television station or cable provider to hometelevision sets. Viewers passively watch the video content played on thetelevision sets, with their only interactivity being channel selection.

[0004] With computers, TV-based video games, and other multimediaenvironments, home users have become accustomed to interacting with thecontent being displayed. It is common for a computer user to selectivelycontrol what, when, and how content is displayed. For example, acomputer user perusing a CD-ROM program on the Civil War might select atopic on Robert E. Lee. The computer user can use a mouse or otherpointing device to page through various screens portraying the life ofthe General, read textual descriptions of Lee's achievements, click onicons to activate audio and video clips of reenacted war scenes. Thewhole experience is interactive.

[0005] The Internet offers an interactive multimedia environment. Withthe swift expansion and popularity of the Internet, many users arebecoming familiar with “surfing” the World Wide Web (“Web” or “WWW”) tolocate information, conduct business transactions, obtain weatherforecasts, conduct banking and other financial matters, shop formerchandise, and so forth. The user can locate and download files inessentially any data format (video, audio, graphical, text, etc.) fromessentially anywhere in the world.

[0006] With such advances in interactive entertainment, there has beensome effort to enhance traditional television video programming withsupplemental content. One approach is to support a television broadcastwith supplemental content made available on the Internet. As an example,MSNBC news, a cable news network jointly established by MicrosoftCorporation and NBC, offers a 24-hour cable news program with concurrentpostings at its Web site containing information supplementing the cablebroadcast.

[0007] Many industry pundits believe that in the future the supplementalcontent will be offered concurrently with the broadcast televisionprogram on the same display device. For example, a home viewer watchingan episode of Gilligan's Island on the television set might be able toaccess target resources on the Internet which relate to Gilligan'sIsland, or other matters, from the same television set. It would bebeneficial to present or make available the supplemental content atspecific time slots in the episode. When Gilligan ruins the Skipper'shat, for example, a producer might want to display a hyperlink to a hatretailer for a brief interval while the episode continues to run. Theuser then has an opportunity to activate the link to call up the hatretailer web site and shop for a new hat.

[0008] One primary problem to enhancing traditional broadcast videoprograms concerns synchronizing presentation of the enhancing contentwith specific junctures or scenes in the video programs. In many cases,the enhancing content is not linked or associated with the video programin any useful manner. In the Gilligan example, a hyperlink to the hatretailer has no built in association to the video program Gilligan'sIsland. To compound the problem, the enhancing content will most likelynot be provided from the same source as the video program.

[0009] The inventor has conceived of a technique for synchronizingpresentation of the enhancing content with specific scenes in the videoprograms.

SUMMARY OF THE INVENTION

[0010] This invention concerns a system and method for synchronizingenhancing content with primary content, such as broadcast televisionprograms, using the closed captioning script of the primary content. Inthis manner, enhancements to the primary content are timely introducedat preselected phrases or raw data strings of the closed captioningscript, and hence at desired junctures of the primary content.

[0011] According to one aspect of the invention, a producer of enhancingcontent obtains the closed captioning script for a video program, suchas a traditional broadcast television show, or a cable movie, or aprogram recorded on video cassette. The producer uses a parsingapplication to parse the closed captioning script to identify one ormore key phrases. Preferably, the parser returns a set of uniquephrases, with each phrase having the same number of words, or a uniquedata character string, with each character string having the same lengthof characters.

[0012] The program enhancement producer decides at what points in thevideo program to introduce enhancing content. As possible examples, theenhancing content might be a hyperlink to a target resource on theInternet, or manipulation of the video window with concurrentintroduction of text or graphical data, or launching an application.After the program enhancement producer outlines the placement of theenhancing content within the video program, the producer uses anauthoring computer to associate supplemental data (e.g., a URL, a filename, etc.) used to activate the enhancing content with specific keyphrases or character strings of the closed captioning script thatcorrespond to the desired points in the program. The authoring computercreates a key phrase data file which contains a listing of the keyphrases or character strings and their association to the supplementaldata.

[0013] The key phrase data file is delivered to viewer computing units(e.g., personal computer, television with set top box, etc.) at users'homes. For example, the data file can be delivered over the broadcastnetwork used to transport the video program, over the Internet or othernetwork, or through the mail on a storage disk.

[0014] When the program is played, a viewer computing unit equipped withthe key phrase data file for that program begins to monitor the closedcaptioning script. The viewer computing unit has a key phrase detectorto detect the key phrases or character strings listed in the key phrasedata file. In one implementation, the key phrase detector is a parserwhich examines sequential groupings of words or characters and comparesthe groupings with the key phrases in the data file to identify the keyphrases. Upon detection of a particular key phrase, the key phrasedetector accesses the key phrase data file to retrieve the supplementaldata associated with the particular key phrase. The viewer computingunit executes an enhancement action according to the supplemental datato synchronize the enhancement action with scenes in the video program.

[0015] According to one implementation, the key phrase detectormulticasts the enhancement action to a multicast address. A programenhancement listener listens to the multicast address to receive andhandle any enhancement action supported by the supplemental data. Thelistener can be implemented as an ActiveX™ control embedded in acontainer, such as an HTML page, or as an application. Usingmulticasting as an interprocess communication tool within the viewercomputing unit effectively decouples the key phrase detection activityfrom the content enhancement activity, allowing the content enhancementproducers to concentrate only on what enhancement actions to make inconjunction with the supplemental data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016]FIG. 1 is a diagrammatic illustration of a system for deliveringboth primary and enhancing content to viewer computing units.

[0017]FIG. 2 is a block diagram of an authoring system for producing anenhanced program in which enhancing content is synchronized to keyphrases in a closed captioning script of the program.

[0018]FIG. 3 is a flow diagram of steps in a method for producing anenhanced program according to one implementation.

[0019]FIG. 4 is a flow diagram of steps in a method for producing anenhanced program according to another implementation.

[0020]FIG. 5 is a diagrammatic illustration of a key phrase datastructure which associates the key phrases of the closed captioningscript with supplemental data used to enhance a program.

[0021]FIG. 6 is a block diagram of a viewer computing unit.

[0022]FIG. 7 is a flow diagram of steps in a method performed by theviewer computing unit for enhancing primary content with enhancingcontent.

[0023]FIG. 8 illustrates the operation of a closed captioning parserresident at the viewer computing unit to identify key phrases in theclosed captioning script.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0024] Conventional broadcast television signals are transmitted in adata format that is used to construct individual television frames. Eachframe contains video image data (i.e., pixel data) used to create imageson the television screen and non-image information used to control thepresentation of the image, such as synchronization and timinginformation. The non-image information is primarily contained within thefirst twenty-one lines of the television frame. These upper twenty-onelines are referred to as the Vertical Blanking Interval (VBI), and arenot displayed on the television. Instead, these lines coincide with thevertical refresh period in which the cathode ray tube of the televisionset is moved from the lower right hand corner of the screen uponcompletion of one frame to the upper left hand corner of the screen forcommencement of the next frame.

[0025] Closed captioning data is broadcast in line 21 of the VBI. Theclosed captioning data provides a textual script of the audio content inthe program. The textual script can be displayed concurrently with theprogram to assist a deaf or hearing impaired audience understand thecharacter dialog or narration.

[0026] The Television Decoder Circuitry Act of 1990 requires that alltelevisions made after 1993 of a size 13 inches or larger be equippedwith closed captioning decoding circuitry. Newer video cassetterecorders are also designed to capture and replay the closed captioningdata for a closed captioning enabled program. As a result, closedcaptioning data is becoming increasingly prevalent on broadcasttelevision shows, syndicated shows, cable movies, and pre-recordedprograms on video cassettes or digital video disks. The NationalCaptioning Institute in Washington, D.C. estimates that nearly 100% ofprime time television shows and between 60%-70% of syndicated shows haveclosed captioning.

[0027] An aspect of this invention concerns exploiting closed captioningas a means to synchronize the presentation of enhancing content tospecific dialog sequences in a program.

[0028]FIG. 1 shows a system 20 which provides an exemplary context forimplementing aspects of the invention. System 20 provides a datadelivery architecture for delivering primary content along withsupplemental or enhancing content to multiple viewer computing units, orclients 22(1), 22(2), . . . , 22(L), . . . , 22(M). The primary contentis the main program to be presented to the viewer, and it supports aclosed captioning script. One notable example of primary content iscontinuous video data. Broadcast television shows, transmitted cableshows, pre-recorded movies on video cassettes or digital video disks(DVDs), live media feeds (e.g., news, sports, debates, etc.), and thelike are all examples of primary content in the form of continuous videoprograms that are often closed captioning enabled.

[0029] The clients 22(1)-22(M) can be implemented in a number of ways,including desktop computers, laptop computers, and computer enhancedtelevision units (e.g., a television with a set top box, a televisionwith set top box functionality built in, etc.). An exampleimplementation in which a client is constructed as a broadcast-enabledpersonal computer is described below with reference to FIG. 6.

[0030] In the FIG. 1 implementation, the primary content is delivered tothe clients 22(1)-22(M) in one of two ways. The first way is tobroadcast the primary content over a conventional broadcast network 24(e.g., RF, satellite, cable, microwave, etc.) from a broadcast stationor cable headend 26. This is the typical approach to deliveringtelevision and cable shows to audiences.

[0031] A second way is to deliver the primary content over a datanetwork 28, such as the Internet, to a broadcast center 30, which thendelivers the primary content over a unidirectional network 32 to theclients 22(1)-22(M). The data network 28 can be implemented in a numberof ways, including wire-based technologies (e.g., fiber optic, cable,wire, etc.) and wireless technologies (e.g., satellite, RF, microwave,etc.). The data network 28 can further be implemented using variousavailable switching technologies (e.g., ATM (Asynchronous TransferMode), Ethernet, etc.) and different data communication protocols (e.g.,TCP/IP, IPX, etc.). In such protocols, the data is packaged inindividual, fixed byte-size packets which are transmitted separatelyover the data network. The unidirectional network 32 can be implementedin a variety of ways, including conventional networks such as satellite,radio, microwave, cable, and so forth. The broadcast network 32 mightalso be implemented as a network which supports two-way communication,but can be used for unidirectional multicasting from the broadcastcenter 32 to the clients 22(1)-22(M) simultaneously.

[0032] For the second approach, one or more content servers 34(1)-34(K)serve the primary content over the data network 28 to the broadcastcenter 30. Content servers used to serve the primary content might beconfigured, for example, as continuous media file servers which servedata files at a constant data rate. An exemplary construction of a fileserver comprises a disk array of storage disks, with the data filesstriped across the storage disks, and one or more servers whichcooperate together to serve the data files from the storage disks.

[0033] The broadcast center 30 includes a router 36, a signal generator38, and a broadcast transmitter 40. The router 36 is coupled to the datanetwork 28 to receive the content served from the content servers34(1)-34(K). The router 36 is a final node of the data network 28 inwhich data communication is bi-directional to that point andunidirectional past that point. The router 36 is preferably configuredas a bridge-router between the traditional data network 28 and thebroadcast network 32. A bridge-router is capable of supporting video andaudio broadcast transmission. Data is received at the router 36 andconverted from the network packet format to a format appropriate forbroadcast transmission. The signal generator 38 generates a broadcastsignal with the data embedded thereon to carry the data over thebroadcast network 32. The broadcast signal is passed to the transmitter40 where it is broadcast over the broadcast network 32 to the clients22(1)-22(M).

[0034] System 20 also allows for the delivery of enhancing content tothe clients 22(1)-22(M). The enhancing content is used to enhance theprimary content. The enhancing content can be supplied to the clients22(1)-22(M) over the broadcast networks 24, 32, or over the data network28 from the content servers 22(1)-22(K). One or more of the contentservers 34(1)-34(K) are used to serve the enhancing content, which maybe in the form of audio, video, animation, bit maps or other graphics,applications or other executable code, text, hypermedia, or othermultimedia types. As an exemplary implementation, the content servers22(1)-22(K) used to serve the enhancing content are implemented aspersonal computers or workstations running a multitasking, disk-basedoperating system, such as Windows® NT from Microsoft Corporation.

[0035] As an alternative, the enhancing content might reside on astorage medium at the viewer's home, such as on a computer disk or aCD-ROM, which can be accessed during the playing of the primary content.

[0036] One example of enhancing content is a hyperlink that is timelydisplayed at a certain juncture of the primary content to allow a viewerto activate the hyperlink and access a target resource on the Internetwhich has additional or related information to the primary content. Forinstance, suppose the primary content is a television episode ofSeinfeld, in which the character Jerry Seinfeld is wearing Levi® jeans.When Seinfeld first appears in a scene, a content enhancing producermight wish to timely introduce a URL (Universal Resource Locator) to aWeb Page for Levi Strauss Corporation that has information about theparticular Levi® jeans that Seinfeld is wearing.

[0037] Another example of enhancing content is to manipulate the mannerin which the primary content is being displayed. For instance, a contentproducer might wish to modify the video window in which the primarycontent is being displayed and concurrently introduce text or graphicaldata alongside or overlaid atop the primary content. One possibleimplementation is to launch an Internet Web browser which presents ahypermedia document, such as an HTML (HyperText Markup Language) page.Within the hypermedia document, the producer can specify display layoutinstructions prescribing how the enhancing content and primary contentare to appear in relation to one another when displayed. A detaileddescription of this implementation is provided in co-pending applicationSer. No. 08/623,610, entitled “Interactive Entertainment System ForPresenting Supplemental Interactive Content Together With ContinuousVideo Programs,” filed Mar. 29, 1996, in the names of Dan Shoff, ValerieBronson, Joseph Matthews, III, and Frank Lawler, and assigned toMicrosoft Corporation. This application is hereby incorporated byreference.

[0038] Another example of enhancing content is to launch an application,such as a Web browser, or a word processing application, or electronicmail. For instance, suppose the primary content is a politicaldiscussion program. Near the end of the program, the producer would liketo gather feedback from the viewing audience regarding the issuesdiscussed on the show. Accordingly, the producer might timely launch anelectronic mail program at a specific point near the end of the programto present a new mail message with the “To” address filled in. Theviewer can add comments and send the e-mail message to the program'sproducer for immediate feedback.

[0039] An aspect of this invention concerns a technique forsynchronizing the enhancing content with the primary content in a mannerwhich is independent of how and when the enhancing content, or theprimary content, are delivered to the viewer computing units. Thetechnique takes advantage of the closed captioning script as the timingmechanism for synchronizing introduction of the enhancing contentrelative to the primary content.

[0040] In general, a content enhancing producer obtains the closedcaptioning script of the primary content and relates key phrases orunique character strings from the script to supplemental data used toactivate the enhancing content. Then, while the primary content isplaying, the clients monitor the closed captioning dialog to detect thekey phrases or character strings and to activate the enhancing content.In this manner, the enhancing content is presented at specific dialogsequences monitored in the closed captioning script, which coincide withparticular junctures in the primary content.

[0041]FIG. 2 shows a computerized authoring system 50 which can be usedby a content enhancing producer to create enhanced programs in whichenhancing I content is synchronized with scenes in a video program orother primary content. Authorizing system 50 is shown implemented as apersonal computer or workstation having a processor 52, volatile memory54 (e.g., RAM), and program memory 56 (e.g., hard disk, floppy disk,flash, ROM, etc.). An electronically readable version of the program'sclosed captioning script 58 is loaded into the program memory 56. Aparser 60 is shown implemented as a computer software program stored inmemory 56 and executable on the processor 52.

[0042]FIG. 3 shows a method for producing an enhanced program using theauthoring system 50. The producer first obtains a copy of the closedcaptioning script for a program and loads the closed captioning script58 in the authoring computer 50. At step 70 in FIG. 3, the parser 60parses the closed captioning script 58 to identify one or more keyphrases. The parser 60 is configured to compile a set of unique keyphrases. Preferably, each key phrase has the same number of words.Consider the following example of a greeting dialog between twocharacters as presented in a fictional closed captioning script:

[0043] Hi, how are you?

[0044] Oh, hi how are you?

[0045] I'm fine thanks.

[0046] The parser 60 first determines the minimum number of wordsrequired to create unique phrases by trying different word lengths andchecking for duplication. In this example, five word phrases are used.Phrases having a four or less word length result in duplication. Forinstance, the four word phrase “hi how are you” is duplicated in theshort dialog.

[0047] Once the minimum length is determined, the parser 60 returns aset of unique phrases with each new phrase starting at sequential wordsin the dialog. From the above greeting dialog, the parser 60 returns thefollowing set of unique phrases:

[0048] hi how are you oh

[0049] how are you oh hi

[0050] are you oh hi how

[0051] you oh hi how are

[0052] oh hi how are you

[0053] hi how are you i'm

[0054] how are you i'm fine

[0055] are you i'm fine thanks

[0056] The parser 60 can also be configured to create unique characterstrings from the raw closed captioning data. For instance, for the text“Hi, how are you?”, the raw data might be “@!)(HI&,%!HOW˜!@#ARE&!!YOU”.The non-alphanumeric characters are control characters which prescribethe positioning and formatting information. The parser 60 creates uniquecharacter strings that have the same character length. Rather thankeying on the words themselves, the raw character stream is monitored tofind the unique character strings.

[0057] For purposes of this disclosure, the term “phrase” is meant toinclude both word sequences of text (e.g., “oh hi how are you”) andcharacter strings of raw data (e.g., “!!OH@!)(HI&,%!HOW˜!@”).

[0058] After compiling the key phrases, the parser 60 outputs a filecontaining the set of unique phrases.

[0059] It is noted that, rather than the parser 60, other types of keyphrase identifiers can be used to identify key phrases in the closedcaptioning script. For example, the processor might be directed tosearch the closed captioning script for a single word, or a set ofwords, spoken by one or more characters to which the producer wants tointroduce enhancing content.

[0060] At step 71 in FIG. 3, the producer determines at which point inthe program to insert enhancing content. Suppose, for example, theauthor wants to display a hyperlink to a Web site for a greeting cardcompany during the greeting scene in the video program. The author mightchoose the phrase “oh hi how are you” as the trigger for displaying thehyperlink. Accordingly, the author uses the authoring computer 50 toinsert supplemental data in the form of a URL at the appropriate placein the closed captioning script. The revised script might read:

[0061] Hi, how are you?

[0062] Oh, hi how<www.greetingcardco.com>are you?

[0063] I'm fine thanks.

[0064] After the producer has added all the enhancing content, theauthoring computer 50 extracts the supplemental data and associates themwith the corresponding key phrases (step 72 in FIG. 3). In this example,the authoring computer 50 retrieves the URL “www.greetingcardco.com”from the parser output file and links or otherwise associates it withthe phrase “oh hi how are you” as follows:

[0065] oh hi how are you=www.greetingcardco.com

[0066] As another example, suppose the author wants to launch anapplication at the response “I'm fine thanks.” The authoring computer 50might create the association:

[0067] are you i'm fine thanks=application.exe

[0068] At step 73 in FIG. 3, the authoring computer 50 creates a keyphrase data file which contains the supplemental data associated withthe key phrases. This key phrase data file, referenced as number 62 inFIG. 2, is stored in memory 56 during creation. The key phrase data fileis subsequently given to the viewer computing units 22(1)-22(M) for usein decoding the closed captioning script when the primary content isplaying to enhance the primary content.

[0069] The FIG. 3 method creates a data file containing a list of allunique key phrases in the closed captioning script. In anotherimplementation, the authoring computer 50 automatically creates areduced set of key phrases, on an “as needed” basis, which correspond tospecific discrete points in the script for introduction of enhancingcontent.

[0070]FIG. 4 shows the alternative method for producing an enhancedprogram using the authoring system 50. As before, the content enhancingproducer obtains a copy of the closed captioning script 58 and loads itin the authoring computer 50. At step 74, the producer reads through theclosed captioning script (and perhaps, concurrently views the program)and identifies spots in the script for introducing enhancing content.The producer might identify such spots by simply clicking a mouse buttonat the particular place as the script is running. In response, theauthoring computer marks the identified spot with an entry in a separatedata file (step 75). The entry specifies the byte offset from the startof the script, as follows:

[0071] byte offset into script file<tab>{place-holder 1}

[0072] next byte offset into script file<tab>{place-holder 2}

[0073] This new place holder data file contains references to a set ofdiscrete points in the closed captioning script at which the producerintends to introduce enhancing content. The place-holders aremonotonically incremented for each entry, and will eventually bereplaced with the supplemental data for the enhancing content, such as aURL, trigger, or application name.

[0074] At step 76 in FIG. 4, the parser 60 determines a minimum phraselength need to uniquely identify phrases at the discrete points of thescript. For a given point, the parser returns a minimum phrase lengthnecessary to uniquely identify that point in the script. The parser thendetermines an overall minimum phrase length based on the minimum lengthsfor each point (step 77 in FIG. 4). That is, the minimum phrase lengthfor the script is equal to the maximum phrase length for any given pointentry in the place holder file. The minimum phrase length may beexpressed in terms of a number of words, or a number of characters, or anumber of bytes.

[0075] Using the place holder file and minimum phrase length, theauthoring computer traverses the closed captioning script andautomatically identifies key phrases of the prescribed length at theselected spots in the script (step 78 in FIG. 4). For each entry in theplace holder file, the authoring computer skips to the byte offset inthe script, and identifies a key phrase having the minimum phrase lengthat that particular spot. This key phrase is then associated with theplace-holder, or supplemental data related to the place holder. As aresult, the authoring computer automatically generates the key phrasedata file 62 containing the key phrases associated with the supplementaldata (step 79 in FIG. 4).

[0076]FIG. 5 shows an exemplary data structure for the key phrase datafile 62. The data structure 62 includes key phrase data fields 80(1),80(2), . . . , 80(S) which contain key phrases 0-S that have beenabstracted from a closed captioning script for use as enhancing contenttriggers. It is noted that the authoring computer might compute hashesof the key phrases to reduce the size of the data file and improvematching speed at the client.

[0077] The data structure 62 also has supplemental data fields 82(1),82(2), . . . , 82(S) which contain supplemental data 0-S for enhancingthe primary content. The key phrase data structure 62 is organized in amanner that associates supplemental data 0-S in the supplemental datafields 82(1)-82(S) with corresponding key phrases 0-S in the key phrasedata fields 80(1)-80(S). The key phrase data structure 62 also has aminimum phrase length data field 84 which contains the minimum number ofwords or minimum number of characters (or data bytes) in each key phrase0-S.

[0078] According to the above implementations, the supplemental data isnot actually added to the closed captioning script. Rather, the authorcreates a separate data file 62 that associates key phrases appearing inthe closed captioning script with supplemental data used to callenhancing content. This is advantageous because the enhanced contentproducers can be independent of the producers of the primary content.The enhanced content producers merely need a copy of the closedcaptioning transcript to aid them in creating the enhancing content, butthey do not ever change or modify the closed captioning transcript.Hence, they can operate independently of the primary content producers.

[0079] Another advantage is that the closed captioning script is almostalways carried with the primary content to the end viewer, even thoughother lines of the Vertical Blanking Interval may be stripped awayduring rebroadcast. The enhancing content can be delivered independentlyof the primary content and synchronized at the viewer computing unitusing the closed captioning script which accompanies the primarycontent. Additionally, most modern video cassette recorders are designedto capture the closed captioning script along with the primary contentduring recording. Thus, even during playback of a recorded program, theclosed captioning script is present to provide the timing mechanism forthe enhancing content.

[0080] The key phrase data file created by the content enhancingproducer is supplied to interested viewers. The key phrase data file canbe delivered in a number of ways, such as broadcasting the file over thebroadcast network used to carry the primary content, or multicasting thefile to a multicast address to which the client listens, or posting thefile at a publicly accessible Web site on the Internet, or supplying thefile to the viewer on a computer disk or other storage medium.

[0081] In another implementation, the supplemental data might beembedded directly in the closed captioning script, rather than beinglinked to the key phrases through a data file. In this implementation,the producer inserts the supplemental data at key phrases in thetranscripts. The supplemental data is formatted to be identified assuch, so that the viewer computing unit does not display thesupplemental data as part of the closed captioning text. Exemplarymodifications that could be made to the closed captioning script includeinsertion of hyperlinks in the closed captioning script, insertion ofreplaceable key-phrases with monikers, and insertion of hidden orspecial characters to which supplemental data is keyed. However,embedding supplemental data directly into the transcript is in some waysless preferable because it requires cooperation between the primarycontent producers and the enhanced content producers. As a result, theenhanced content producers lose some of their freedom to create, whileother enhanced content producers may be prevented from participating atall.

[0082] In another embodiment, the authoring computer system might beconfigured to write out the entire closed captioning script along withtiming information of the script relative to the video content (e.g.,frame numbers). The closed captioning is stored as a text file, and thevideo is stored in a digital format, such as an AVI file. For each wordin the closed captioning transcript, the authoring system writes theassociated video frame number. The producer can then select places toinsert enhancing content using the video itself, with the authoringsystem linking the enhancing content to the appropriate phrases in theclosed captioning script.

[0083] Now, suppose that a viewer decides to watch a particular primarycontent that is capable of being enhanced. For purposes of continuingdiscussion, suppose that the primary content is delivered over abroadcast network and that the viewer has already been supplied with thekey phrase data file 62.

[0084]FIG. 6 shows a viewer computing unit or client 22(1) constructedas a broadcast-enabled computer according to one exemplaryimplementation. It includes a central processing unit 90 having aprocessor 92 (e.g., ×86 or Pentium® microprocessor from IntelCorporation), volatile memory 94 (e.g., RAM), and program memory 96(e.g., ROM, disk drive, floppy disk drive, CD-ROM, etc.). The client22(1) has one or more input devices 98 (e.g., keyboard, mouse, etc.), acomputer display 100 (e.g., VGA, SVGA), and a stereo I/O 102 forinterfacing with a stereo system.

[0085] The client 22(1) includes a broadcast receiver 104 (e.g.,satellite dish receiver, RF receiver, microwave receiver, multicastlistener, etc.) and a tuner 106 which tunes to appropriate frequenciesor addresses of the broadcast network 24, 32 (FIG. 1). The tuner 106 canbe configured to receive the primary content in a particularized format,such as MPEG-encoded digital video and audio data. The client 22(1) alsohas a modem 108 which provides access to the data network 28 (FIG. 1).For other implementations, the modem 58 might be replaced by a networkcard, or an RF receiver, or other type of port/receiver which providesaccess to a compatible data network.

[0086] The client 22(1) runs an operating system (not shown) whichsupports multiple applications. The operating system is preferably amultitasking operating system which allows simultaneous execution ofmultiple applications. The operating system employs a graphical userinterface windowing environment which presents the applications ordocuments in specially delineated areas of the display screen called“windows.” One preferred operating system is a Windows® brand operatingsystem sold by Microsoft Corporation, such as Windows® 95 or Windows® NTor other derivative versions of Windows®. It is noted, however, thatother operating systems which provide windowing environments may beemployed, such as the Macintosh operating system from Apple Computer,Inc. and the OS/2 operating system from IBM.

[0087] One example implementation of a broadcast-enabled PC is describedin a co-pending U.S. patent application Ser. No. 08/503,055, entitled“Broadcast-Enabled Personal Computer,” filed Jan. 29, 1996 in the namesof Gabe L. Newell, Dan Newell, Steven J. Fluegel, David S. Byrne,Whitney McCleary, James O. Robarts, Brian K. Moran; William B.McCormick, T. K. Backman, Kenneth J. Birdwell, Joseph S. Robinson,Alonzo Gariepy, Marc W. Whitman, and Larry Brader. This application isassigned to Microsoft Corporation, and is incorporated herein byreference.

[0088] The client 22(1) is illustrated with two software programs: aclosed captioning parser 110 and an HTML page with an embedded multicastlistener 112. Each program is stored in program memory 96, loaded intovolatile memory 94 when launched, and executed on the processor 92. Thekey phrase data file 62 is also shown stored in the program memory 96.The closed captioning parser 110 is configured to monitor the closedcaptioning script as the video program is played to detect the keyphrases listed in data file 62.

[0089]FIG. 7 shows a method for enhancing the primary content withsupplemental data. At step 120, the client prepares for playing anenhanced program by initializing the closed captioning parser 110 to theminimum word length or character count of the key phrases. Thisinitialization sets the parser 110 to monitor the closed captioningscript in successive groups of words equal to this word length orcharacter strings equal to the character count. The client obtains thekey phrase word length from the minimum phrase length data field 84 ofthe data structure 62 (FIG. 5).

[0090] At step 122 in FIG. 7, the client 22(1) plays the primary contenton the display 100. In this example, the receiver 104 receives theprimary content from the broadcast network and passes it to the centralprocessing unit 90. The CPU 90 strips the non-video data in the VerticalBlanking Interval and passes the video data to video display drivers forpresentation on the display 100. The primary content contains a closedcaptioning script which is contained as part of the Vertical BlankingInterval. The text of the closed captioning script may or may not bedisplayed on display 100 concurrently with the primary content incustomary fashion.

[0091] As the primary content plays, the closed captioning parser 110monitors the successive word groups in the closed captioning script(step 124 in FIG. 7). The parser 110 compares each word group with thelist of key phrases in the key phrase data fields 80(1)-80(S) of thedata structure 62 (step 126 in FIG. 7). If no match occurs (i.e., the“no” branch from step 128), the parser 110 continues to the next wordgroup (step 130). It is noted that, if the data file contains hashvalues of the key phrases, the parser 110 may need to compute hashes ofeach word group for comparison to the stored hash values.

[0092]FIG. 8 illustrates this process for parsing the closed captioningscript using the greeting dialog from the examples given above. Theparser 110, which is initialized to a word length of five, examinessuccessive groups of five words in the closed captioning script 140. Thefirst group 142 is the five word phrase “hi how are you oh.” This phraseis compared to the key phrases in data structure 62 and no match occurs.Thus, the parser 110 moves to the next group 144 which provides the fiveword phrase “how are you oh hi.” Once again, the phrase is compared tothe key phrases and no match occurs. The process is repeated for thenext group 146, and for successive groups thereafter.

[0093] When a group of words matches a key phrase in the data structure62 (i.e., the “yes” branch from step 128), the parser 110 looks up inthe data file for the supplemental data corresponding to the matched keyphrase. In this manner, the data file is utilized as an associationlook-up table. The supplemental data is associated with the key phrasesthrough the inherent corresponding field arrangement of the datastructure 62. The parser 110 retrieves the supplemental data from thekey phrase data file 62 (step 132 in FIG. 7). In the continuing example,the parser 110 detects the phrase “oh hi how are you” which matches akey phrase stored in data structure 62. The parser then retrieves thehyperlink “www.greetingcardco.com,” which is associated with the phrase,from the data structure 62. The client can now use the supplemental datato activate an enhancing action which enhances the primary content, suchas displaying the hyperlink on the screen along with the primarycontent.

[0094] According to an aspect of the invention, the client employsmulticasting as an interprocess communication technique. At step 134 inFIG. 7, the parser 110 sends the supplemental data to a multicastaddress monitored locally by the client. The parser 110 then continuesto the next word group (step 130 in FIG. 7).

[0095] Meanwhile, the HTML container with the program enhancementlistener 112 listens to the multicast address to receive thesupplemental data (step 136 in FIG. 7). Upon receiving the supplementaldata, the listener operates on the supplemental data to perform theenhancing action, such as displaying a hyperlink, or launching anapplication, or displaying supplemental data concurrently on the screenwith the primary data (step 138 in FIG. 7). In this manner, theenhancement action is synchronized with the particular scene in whichthe dialog “oh hi how are you” takes place.

[0096] More specifically, the listener is preferably implemented as anActiveX™ control which sits in a container, such as the HTML page, or anapplication. ActiveX™ is a technology developed and marketed byMicrosoft Corporation. ActiveX™ controls are described in detail in abook entitled, Exploring ActiveX, by Shannon R. Turlington, VentanaCommunications Group, Inc., copyright 1996, which is hereby incorporatedby reference. The listener examines the supplemental data received atthe multicast address and decides what action to take.

[0097] The supplemental data can be encoded to indicate its type. Forexample, supplemental data in plaintext means it is a hyperlink (e.g.,www.greetingcardco.com); supplemental data enclosed in quotes means itis an executable file (e.g., “application.exe”); and supplemental dataenclosed in curly brackets means it is a trigger for arbitrary text orgraphical data (e.g., {text}). Depending upon the type of supplementaldata, the listener takes one of the following actions: (1) instructs aWeb browser to bring up a URL, (2) expose the arbitrary message directlyto the container, or (3) launch an application. The type identifiersexplained above are examples, as any arbitrary convention foridentifying data type may be used.

[0098] With the multicast listener and interprocess communicationtechnique, the enhancement action is decoupled from the parser activityfor detecting key phrases. This is advantageous from a productionstandpoint as it permits the enhanced content producers to concentratesolely on the enhancing content to be activated by the supplementaldata, without regard to how and when the parser detects the key phrasesor retrieves the supplemental data.

[0099] The invention is advantageous in that it leverages the existingclosed captioning available in most programs as a timing mechanism forsynchronizing enhancing content with the primary content. With thismethod, the program can be started mid-stream and the enhancing contentfor the remaining portion will still be timely cued at appropriatejunctures of the primary content.

[0100] The implementation described above is explained in the context ofreceiving broadcast programs, such as television shows, from a broadcastnetwork and the enhancing content from content servers on the Internet.This is but one example as other example implementations exist. Forinstance, another example might be to locally supply the primary contentand closed captioning script from a video cassette and the supplementaldata from a separate computer disk, such as a CD-ROM.

[0101] In compliance with the statute, the invention has been describedin language more or less specific as to structural and methodicalfeatures. It is to be understood, however, that the invention is notlimited to the specific features described, since the means hereindisclosed comprise preferred forms of putting the invention into effect.The invention is, therefore, claimed in any of its forms ormodifications within the proper scope of the appended claimsappropriately interpreted in accordance with the doctrine ofequivalents.

1. For use with a viewer computing unit having a processor, a programenhancement listener implemented as computer-executable instructionsstored on a computer-readable medium and executable on the processor todirect the viewer computing unit to: receive supplemental data sent to amulticast address; and initiate an enhancement action based upon thesupplemental data to enhance a video program as the video program isbeing played.
 2. A program enhancement listener as recited in claim 1,wherein the enhancement action comprises activation of a hyperlink.
 3. Aprogram enhancement listener as recited in claim 1, wherein theenhancement action comprises launching executable code.
 4. A programenhancement listener as recited in claim 1, wherein the supplementaldata is associated with at least one key phrase of a closed captioningscript of the video program.
 5. A program enhancement listener asrecited in claim 1, further comprising computer-executable instructionsto direct the viewer computing unit to display the supplemental dataconcurrently with the primary content.
 6. A program enhancement listeneras recited in claim 1, further comprising computer-executableinstructions to direct the viewer computing unit to: present the videoprogram within a hypermedia document; and controlling placement of thevideo program within the hypermedia document using the supplementaldata.
 7. For use with a viewer computing unit having a processor and adisplay, a hypermedia document stored on computer-readable medium andexecutable on the processor for graphical rendering on the display, thehypermedia document containing the program enhancement listener asrecited in claim
 1. 8. A method comprising: receiving supplemental datasent to a multicast address; and initiating an enhancement action basedupon the supplemental data to enhance a video program as the videoprogram is being played.
 9. A method as recited in claim 8, furthercomprising associating the supplemental data with at least one keyphrase of a closed captioning script of the video program.
 10. A methodas recited in claim 8, wherein the supplemental data comprises ahyperlink to a target resource, and the initiating comprises activatingthe hyperlink to the target resource.
 11. A method as recited in claim8, wherein the supplemental data 2comprises executable code, and theinitiating comprises launching the executable code.
 12. A method asrecited in claim 8, further comprising displaying the supplemental dataconcurrently with the primary content.
 13. A method as recited in claim8, further comprising: presenting the video program within a hypermediadocument; and controlling placement of the video program within thehypermedia document using the supplemental data.
 14. A computerprogrammed to perform the method as recited in claim
 8. 15. Acomputer-readable media having computer-executable instructions forperforming the steps of the method as recited in claim
 8. 16. A systemcomprising: means for listening to a multicast address for supplementaldata; and means for initiating an enhancement action based upon thesupplemental data to enhance a video program as the video program isbeing played.
 17. A system comprising as recited in claim 16, whereinthe supplemental data comprises a hyperlink to a target resource, andthe initiating means activates the hyperlink to the target resource. 18.A system comprising as recited in claim 16, wherein the supplementaldata comprises executable code, and the initiating means launches theexecutable code.
 19. A system comprising as recited in claim 16, furthercomprising means for displaying the supplemental data concurrently withthe primary content.
 20. A system comprising as recited in claim 16,further comprising: means for presenting the video program within ahypermedia document; and means for controlling placement of the videoprogram within the hypermedia document using the supplemental data. 21.A viewing unit comprising: a key phrase module to identify one or morekey phrases from the closed captioning script and to associatesupplemental data to the one or more key phrases identified from theclosed captioning script; the key phrase module multicasting theenhancement action to a multicast address; a program enhancementlistener to listen to the multicast address for the enhancement actionand to initiate an enhancement action based upon the supplemental datato enhance a video program as the video program is being played.
 22. Aviewer unit as recited in claim 21, wherein the program enhancementlistener comprises a control embedded in a container.
 23. A viewer unitas recited in claim 21, wherein the program enhancement listenercomprises a control embedded in an HTML page.
 24. A viewer unit asrecited in claim 21, wherein the program enhancement listener comprisesa control embedded in an application.